The Network Stack Trace: Performance diagnosis for networked systems

Transient network stalls that degrade application performance are frustrating to users and developers alike. Software bugs, network congestion, and intermittent connectivity all have the same symptoms—low throughput, high latency, and user-level timeouts. In this paper, we show how an end host can identify the sources of network stalls using only simple counters from its local network stack. By viewing the network stack as a producerconsumer dependency graph and monitoring its activity as a whole, our rule-based expert system correctly identifies which modules are hampering performance over 99% of the time, with false positive rates under 3%. The result is a network stack trace—a lightweight snapshot of the end host’s networking stack that describes the behavior of each application, socket, connection, and interface.